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ABSTRACT 

A coaparative case study to evaluate the 
adainistration of Eleaentary and Secondary Education Act (ESEA) Title 
I prograas vaa conducted in eight s-^ate departaents of education and 
32 school districts. Interviews were suppleaented by review and 
analysis of docuaents and available data at each site. General 
findings included: (1) evaluation was priaarily considered to nean 
achieveaent testing: (2) evaluation was unrelated to other 
ad aini strati ve activities, such as prograa design or change: and (3) 
attention was focused on evolving sophisticated evaluation techniques 
rather than on usinq test results. Soae proaising evaluation 
techniques were used by a few school districts: establishing 
obiectives against which to evaluate the Title I prograa: conducting 
a study of the instructional process within the Title I prograa: and 
atteaptinq to institute a content, input, process, and product (CIPP) 
evaluation aodel. (laplications of this case study findings for other 
federally funded educational prograas were discussed, as well as the 
Title I Evaluation and Reporting Systea). (HH) 
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Evaluation in Compensatory Education Programs: 
Problems, Promising Strategies and Recent Trends 

by 

Pascal D. Forgione, Jr., Bernard A. Kaplan and Martin E. Orland* 



INTRODUCTION 



Since the inception of ESEA Title I in 1965 the Title I statute has re- 
quired States and Local Education Agencies to evaluate the effectiveness of 
their Title I programs. Determining what states and school districts have 
done to meet that requirement and why was one of the explicit purposes of a 
1976-77 comparative case s.ady of ESEA Title I administration in eight states 
and 32 school districts. The study was conducted by the Syracuse Research 
Corporation for the National Institute of Education. This paper reports 
the major findings of the SRC study relative to the evaluation question. 
It is divided into five sections. Section I provides background to SRC's 
comparative case study research approach and its particular applicability to 
understanding grantee approaches to evaluation. Section II highlights the 
study's major findings the most important of which is that evaluations were 
condu^^ted for reporting rather than utilization purposes. In Section III of 
this report some unique and promising Title I evaluation practices in three 
cf the districts studies are described. Sections IV and V attempt to place 
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current perspectives on the study's research findings by positing the 
study's implications for evaluations of other federal education programs 
(Section IV) and by relating the research findings to recent federally induced 
changes in the Title I evaluation system (Section V), 
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I. General Background 

During school year 1976-77 the Syracuse Research Corporation (SRC) in- 
itiated a comparative case study of the federal Elementary and Secondary 
Education Act CESEA) , Title I program focusing on intergovernmental (f ederal- 
state-local) relationships. This policy investigation involved eight state 
departments of education and thirty-two school districts and was reported to 
the National Institute of Education in November 1977 (Goettel et.al., 1977). 
A major component of each of the forty individual case studies was the issue 
of evaluation. The ESEA, Title I legislation requires that states, and school 
districts conduct annual program evaluations of Title I services. 

As public education encounters increasing demands for accountability and 
basic skills education, the experiences of this national compensatory education 
program should be especially noteworthy since both of these elements have been 
prominent in Title I program requirements for several years. Lessons learned 
in implementing Title I evaluations should provide guidance to educational 
evaluators in meeting the program assessment requirements of other federal and 
state evaluation mandates. Moreover, evaluation results are now increasingly 
being regarded as vital to policy and administrative decision making; this view 
of the program evaluation role has been proposed for many of the large-scale 
federally-funded programs, including ESEA, Title I. Thus, the net experience 
of Title I program evaluation endeavors across the states and local districts 
should provide a rich and valuable source of information for the education 
evaluation community. 

The information reported in this paper has been drawn in large measure 
from SRC's eighteen month comparative case study of the administration of 
ESEA, Title I programs. An elite inter\'iewing/data collection technique was 
utilized as part of a series of case studies which were designed to include 
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eight SEAs, thirty-two LEAs (four in each state) and 116 participating Title I 
schools. The interviews were supplemented by careful review and analysis of 
documents and available data sources at the various lev^ilt:. Evaluation was 
one of eight compensatory education issues that were probed In the SRC study. ^ 
Case study participants (totaling approximately 1,100 intec\^iewees , including 
over 170 Title I parents) were carefully sfclt»cted by the Individual case study 
research teams at each level (federal, state, district, and school building). 
Title I and regular school district evaluation personnel, aa well as administra- 
tors, teachers, Title I parents and community represen tarives were interviewed 
at the school and district levels. The issue of evaltiation In compensatory 

education was also examined in interviews with federal ^.nd state compensatory 

2 

education program administrators. 

If we agree with Cronbach that evaluation is political activity performed 
within a social system Ccronbach, 1977), then the comparative case study metho- 
dology used in the SRC study ic? an excellent framework for investigating how 
state and local agencies implement their Title I evaluation efforts in a manner 
consistent with state and local social systems. The SRC comparative case study 
data provides unique understandings of what states and local districts are do- 
ing in Title I evaluations, that is, the context within which ESEA, Title I 
evaluations occur and the processes used in implementing this requirement of 

the law, and the policy variables or factors that facilitate or impede evaluation 
efforts. 

^Generally, each of the SRC case studies describes and discusses four Title I 
funds allocations issues (targeting, eligibility, concentration of funds, and 
comparability) and four Title I program development issues (needs assessment, 
program design, evaluation and parent involvement) . Additional central themes 
discussed in each case included: (a) contextual background; (b) agency struc- 
ture and organization; (c) approaches to Title I management functions; and 
(d) state/local, federal/state interrelationships. 

2 

"The authors wish to acknowledge their indebtedness to their SRC colleagues 
who shared authorship for the eight case study volumes and the final synthesis 
report from which sources the material for this paper is largely drawn. 
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II' General Findings Regardina Title I Evaluation Policy and Practice 

A major finding of the study concludes that evaluation primarily means 
"testing" to both states and local districts; evaluation rarely related locally 
to revisions In program deslgr State and local program evaluation was viewed 
most frequently as a mechanical exercise unrelated to other administrative 
activities. In general, state policy In the evaluation area was generally 
restricted to Insuring that local test score data was submitted to the SEA 
on time each year. Three states that can be characterized as directive, 
especially In programmatic areas made somewhat more assertive efforts In this 
area, but even In these states the evaluative emphasis seemed to be more on 
technique than usage, and on state more than local utility. 

With some exceptions, the general attitude In LEAs was that evaluation 
is essentially a necessary evil, conducted to please someone else, with 
little apparent relevance to local programmatic concerns. In some projects, 
there were active attempts to utilize evaluation data for a variety of purposes, 
but those attempts tended to be a function of the drive and competence of the 
evaluator. There Is some evidence that the overall planning requirements of 
one state and the general emphasis on evaluation In a second helped to create 
environments In which Title I evaluation Initiatives could be legitimized. 
Most of the observed Improvements in local evaluation activity occurred largely 
in the ability of LEAs to provide their SEAs with required information, while 
both the interest in and the capacity of LEAs to use test score results and 
more comprehensive evaluative information for management and program decision 
purposes seemed to exist in only a handful of the thirty-two districts included 
in this study, 

A. State Policy and Practice . In five of the eight states, nearly all 
energy in evalution was devoted to assuring that each district had submitted 
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pre- and post-test data on the Title I participant population. Little if 
any effort was made in these states to encourage the districts to link 
evaluation efforts to program design or to give technical assistance to 
districts in this area. Three states which assumed a more directive state 
posture toward program administration of ESEA Title I had somewhat more 
aggressive evaluation policies, although more attention to evaluation seemed 
called for in each of these states. Two of the three had long emphasized the 
importance of evaluation in overall program design. This message was conveyed 
in one of those states more as an overall area of state emphasis (a statewide 
accountability thrust and formal and informal communications from SEA trained 
staff in the evaluation area) than in any specific state policy. A third 
state was particularly noteworthy for its efforts at the state level in 
evaluating and understanding the effects of Title I. They also frequently 
disseminated this information to districts in an effort to give them a tool 
with which to plan their future activities. 

Local Policy and Practice . Two fundamental themes characterized 
evaluation procedures and activities in the LEAs included in this study. 
First, evaluation was typically synonymous with achievement testing— very 
little attention was given to process evaluation or to the systematic 
evaluation of "affective" program objectives. Second, evaluation was primarily 
done in order to fill out required reports; results rarely were used to 
influence program characteristics. Only about six of the thirty-two LEAs 
across six states were engaged in program evaluation activities that extended 
beyond standardized norm-referenced achievement testing of students or that 
attempted to relate results to program changes. Even in states that made some 
efforts to emphasize the usefulness of evaluations, districts generally viewed 
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Promises to maintain confidentiality of the states and districts visited 
preclude their specific mention in this paper or in the case studv reports 
themselves. 
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thelr states as more preoccupied with technique than with usage, and more 
concerned with evaluation for state-level decision-making than local decision- 
making . 

For many of the local programs, the almost exclusive focus on achievement 
testing (to the exclusion of process evaluation or attention to the outcomes 
of "affective" program components) would seem to be a predictable outcome of 
state and federal evaluation requirements. States played a significant role 
in determining which data would be collected and how. And, as indicated 
above, almost all data required were standardized, norm-referenced test score 
data. Some LEAs, however, went beyond state requirements. One large school 
district, for example, required schools to report information on program 
characteristics, parent participation, auxiliary service components, pupil 
services, and attend;ince services. Another district not only utilized and 
reported to the SEA criterion-referenced test results (rather than standardized, 
norm-referenced tests), but also developed a comprehensive evaluation report 
of all compensatory education services in the LEA including the state program, 
ESAA, Title I tUgrant, and Bilingual Education. These exceptions aside, it 
was clear that state reporting requirements provided the sole rationale and 
design for most local evaluation efforts. Local program personnel thus regarded 
evaluation as a necessary activity conducted for someone else's purposes. 

Another general observation on local evaluation efforts is that attention 
to evaluation was attention to technique, not to usage. Indeed, the techno- 
logical capacity of many school districts— the ability to mount a major testing 
program, compare pre- and post-test results, produce school summary data, pro- 
duce listings of individual pupils by school to submit a Title I evaluation 
report to the SEA in mid-summer, and often to do some reasonably sophisticated 
and interesting analysis of test results— these technological capacities far 
exceeded the capacity and/or willingness of LEA policy-makers and managers 
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to utilize evaluation results as a management tool. State policy was often 
an important factor here. States would frequently comment quite favorably 
about sophisticated evaluation techniques in particular districts without 
questioning how the evaluations were being used to help the district design 
future activities. 
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III. Some Promising Local Practices 

As suggested above, SEA requirements for evaluation generally consisted of 
little more than aggregating standardized, norm- referenced test score data 
accumulated from testing Title I students. Of the thirty- two LEAs Included 
In the study, the study teams Identified three that went beyond this minimum. 
Two of these were in large metropolitan city school districts with fairly 
large research and evaluation staffs at the central office level. The third 
consisted of a medium-sized city district serving also as the state's 
capital city. 

A. District 1.. Evaluat ion Activities . In this large district, each school 
has a designated evaluation chairperson responsible for coordinating evaluations 
and testing programs at the school level (for all programs). Each school was 
expected to establish objectives, or checkpoints, against which it can evaluate 
its progra-n. Also, through a needs assessment activity, the school's present 
status was ascertained. Schools were required to report information on Title 
I program characteristics, parent participation, auxiliary service components, 
pupil services and attendance services. The district's Office of Research and 
Evaluation prepared an annual series of evaluation mini-reports which review 
the outcome or status of the varloiis program components. 

While the state and the district had overlapping standardized testing 
program requirements in the reading and mathematics areas, the uistrlct's 
testing was conducted utilizing a matrix sampling of questions. Thus, data 
were available on state, district and school levels but not for the Individual 
student. A reduction in testing time from three hours to one class period 
was realized as a result. 
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Another Interesting aspect Oi the testing program in this district was 
that only test scores of pupils who had been in school at least eighty percent 
of the possible instructional time were used in computing the evaluation of the 
school^ s program 

The district's Office of Research and Evaluation provided two exemplary 
services to its schools. A computer print-out providing standardized testing 
information from all sources was prepared annually. This information included 
an analysis of the school's student population and offered individual pupil 
achievement scores. The schools could use these data for planning and evaluating 
program thrusts as well as to identify those children who scored below the 
median for inclusion in appropriate Title I components. 

Further, the Office of Research and Evaluation assigned its staff to 
the Title I technical assistance teams so that expertise in this area could 
be shared with area curriculum coordinators, school administrators, school 
faculty members and Title I personnel in the district. In this way, the 
individual schools could develop and implement curriculum procedures based 
on evaluation findings and have the benefit of evaluation staff assistance 
as special program materials are developed, tried out and revised. 

B. District 2., Evaluation Activities . Another large-city district, this 
one employed a staff of professionals probably unique in terms of their 
qualifications and numbers. 

Two types of evaluation are performed in this district relative to Title I. 
The first consists of the traditional assessment of Title I student achievement 
based on a fall and spring check. In addition, similar groups of students not 
participating in the programs were identified and tested. 

The second type of evaluation consisted of a study of the instructional 
process within the context of the Title I program. This district has experi- 



ERiC 



-11- 

mented with a variety of programs in reading and mathematics as well as in 
social studies, science and music. Examples of research questions systematically 
explored and reported on by ^he district's evaluation teams include the 
following; 

1. Was the Title I (reading) program implemented as proposed? 

2. What services were rendered to the Title I (reading) students by 
the team of resource and demonstration teachers? 

3. What type and amount of individualized instruction was characteristic 
of the three programs in Title I reading? 

4. Was the basal curriculum taught? To what extent was the Title I 
reading programs' supplemental instruction integrated with the 
total (basal) curriculum? 

5. To what extent were the reading achievement goals met? 

6. How did the results of the Title I reading program compare with 
the results of other district reading programs? 

7. Did the Title I reading programs differ significantly in pre-post 
gain trends as measured by standardized tests, when considered by 
school, by grade, by attendance in the program, and across entire 
programs? 

8. Did the Title I reading programs differ significantly in longitudinal 
(cross-sectional and cohor^) trends for the past three years?' 

Evaluation reports, thoroughly prepared and carefully documented, and based 
on the findings with respect to the above questions, were shared with program 
managers, district administrators and the district-wide PAC. (The final 
evaluation reports in a recent year, for tlie Title I programs in reading, 
in mathematics, and in social studies averaged 96 pages each.) The district 
has used these reports to determine the effectiveness of alternative program 
components, on a cross-comparative basis, on a cost-benefit basis, and as 
compared with teacher/student satisfaction and performance. All of these were 
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considered by district officials in making decisions concerning program 
continuance, modification or elimination. (It should be noted that this 
process did not apply solely to the district's Title I programs; the procedure 
was routinely performed for all of the district's experimental and/or project- 
type programs.' 

^' District 3.. Evaluation Activities . One of the most comprehensive, though 
still largely unproven, efforts at systematically building evaluation pro- 
cedures into the ongoing management process at the district and school levels 
was underway in this district at the time of the site visit. Here, there 
was an attempt to operationlize the CIPP (context, input, process and product) 
evaluation model in each one of the LEA's schools, with particular emphasis 
on those receiving Title I funds. 

In this medium-sized city school district, the staff from the central 
evaluation office launched a comprehensive in-service training activity de- 
signed to instruct Title I teachers and principals not only in how to administer 
tests, but also in how to use the results to improve instruction. Attention 
was also given to interpreting the results of the testing program to parents 
and members of the PACs. The evaluation here would soon rely heavily on 
criterion-reference tests, under construction during the period of our study. 
Tlie district has annually published a report of its comprehensive evaluation 
of state and federal compensatory education programs. This report showed the 
extent to which district instructional objectives were being mastered by 
pupils participating in one or more of the compensatory education programs as 
a group as well as on the individual programs. The report also examined schools 
as the unit of inquiry and for the year of our study the conclusion was that 
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the available data, "reinforce the idea that the school (rather than program) 
is the meaningful unit to look at." 

Thus, the breadth and the focus of the Title I evaluation in this LEA were 
a result of district officials' intention to use evaluation information as a 
management tool for improving the implementation of compensatory education and 
other programs in the district. 

Summary and Conclusions > Overall, these instances of evaluation practice 
are perhaps more significant because they depart from the norm rather than for 
their levels of sophistication relative to evaluation design or execution. This 
is not to detract from the programs for in terms of Title I evaluation practice, 
they appear to be distinguished by the very fact that they exist in their current 
form. 

The three LEAs in which these approaches were in operation shared these 
commonalities : 

(1) The district's evaluation staff, as a separate unit at the central office 
worked closely with the agency's Title I adirinistrators and program officers, 

(2) There was considerable interest, if not actual pressure, emanating from 
the district's administration (and as a result from the research office) for more 
than a routine aggregation of test scores and statistics relative to their Title I 
program evaluation. 

(3) The desire to evaluate Title I programs extended to other experimental, 
project-type programs in the district. The evaluation thrust is not restricted 
to Title I programs. 

(4) Considerable, often impressive cadres of expertise and talent were 
evident in the district's central office research evaluation staffs. And in two 
of the three cases, this was further exemplified by substantial numbers of these 
specialists supported by local agency funds. 
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IV. Implications of SRC Case Study Findings for 
Other Federal Aid Programs in Education 



Although ESEA Title I is the largest federally supported program for 
public education in the country it is by no means the only one. A number 
of other federal education programs are currently operating in the nation's 
schools including programs supported by the Emergency School Assistance Act, 
Follow Through, The Vocational Education Act, The Bilingual Education Act 
and most recently the Education for all Handicapped Children Act. Some of 
these programs have recently attempted to implement new stricter evaluation 
requirements in order to enhance program accountability. For example, Sec- 
tion 112 of the 1976 Vocational Education Amendments requires states for the 
first time to evaluate the effectiveness of each of their grantee programs. 
Similarly the 19 78 Amendments to the Bilingual Education Act specifically 
require that grantee programs provide measurable goals for determining when 
children no longer need assistance and include specific evaluation plans 
consistant with U'SOE guidelines. (Section 721 (a) P.L. 95-561). Since 
Title I was the first program to mandate specific evaluation requirements, it 
seems reasonable to surmise that some of the trends that have characterized 
evaluation implementation in that program will tend to emerge in these other 
programs as well. The following tendencies, based on the SRC Case Study ex- 
perience, should therefore bear careful watching: 

1. An increased pressure on grantees to demonstrate positive returns on 
the invested dollar and in the words of one state Title I administrator 
to 'load their programs for success" i.e., to select program participants 
who are likely to show achievement growth. 
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2. A tendency to narrow the evaluation focus to one or two outcome 
variables to be measured and to require a reporting on these 
measures even when the programs in operation cover different 
areas . 

3- A heavier reliance on testing, particularly standardized norm- 
referenced tests measuring growth in the cognitive domain, 

4. A tendency toward increased fragmentation of services to students 
as programs mandate demonstrations of discrete isolable impacts, 
and with this a tendency for the evaluators of these discrete 
programs to be physically and substantively isolated from the 
agency *s core evaluation and research staff. 

5. An increased emphasis on conducting and reporting on summative 
evaluations without consequent attention to furmative evaluation 
activities. 

6. A tendency to look for short-term gains in student performance 
without consequent attention to long term growth. 

A tendency to leave evaluation to the hands of professional 
evaluators and to limit the evaluational role of others (such as 
program staff, parents and classroom teachers) both in designing 
the evaluation and in utilizing its findings for program improvement 
purposes . 
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V, Recent Title I Evaluation Developments and their Implications ; 
The Title I Evaluation and Reporting System 

Because ESEA Title I was the first federal education program to 
require specific grantee evaluation requirements, the evaluation system 
for this program has had some time to develop and change. The years 
since the SRC case studies have been marked by Increasing efforts of 
the Federal Government to Improve grantee evaluations most notably 
through mandating a new Title I Evaluation and Reporting System for 
grantee agencies (known as TIERS), and by funding Technical Assistance 
Centers (TACs) In each of the 10 HEW regions to assist grantees In 
their evaluation activities. It therefore seems appropriate to 
conclude this paper by to describing and analyzing these new elements 
in the Title I evaluation picture, paying particular attention to the 
question of their current and potential impacts on altering the 
general conditions observed in the SRC site visits^. The discussion 
will begin by briefly recounting the history of the newly mandated 
Title I evaluation models and the TACs, and will proceed from that 
point to assessing actual and likely future Impacts from the 
perspective of the prior research findings of the SRC case studies. 



Since the NTS Research Corporation is one of the designated TACs 
(responsible for HEW Region 3) discussions with Dr. Orland's colleagu 
plus some of his personal experiences in rendering Title I technical 
assistance to LEAa have proven extremely useful in reporting on this 
question. 
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A. Mandated Models and TAG Backgroun d 

The education amendments of 1974 set the stage for the imposition 
of mandated local evaluation models in Title I and the creation of 
Title I Technical Assistance Centers. Section 151 part (d) of the 
amendments required the USOE Commissioner to provide models for Title 
I evaluation, "which shall include uniform procedures and criteria to 
be utilized by local education agencies, as well as by the State 
agency in the evaluation of such (PL 93-380, Sec. 151, part (d)). 
Part (f) of this section required that the models "specify objective 
criteria which shall be utilized in the evaluation of all programs and 
shall outline techniques .. .and methodology. . .for producing data which 
are comparable on a statewide and nationwide basis." (PL 93-380, Sec. 
151, Part (f)). 

The U.S. Office of Education interpreted these sections as 
requiring that they develop Title I evaluation models that could 
produce aggregable results of Title I students' cognitive achievement 
gains across states and school districts, while still allowing LEAs 
significant local program flexibility in determining program 
substance and evaluation instrumentation. The RMC Corporation under 
contract with USOE developed three such models; a norm reference 
model (Model A), a comparison group model (Model B) and a special 
regression model (Model C). The models are all similar in that they 
posit expectations of what a Title I student's achievement level would 
be vdthout Title I services, thus allowing for a comparison between 
actual achievement with such services and hypothesized performance 
without it. They differ in the way in which a no treatment 
expectation is derived. In the norm reference model it is 
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derived from normative test data of students having the same 
percentile pre-test status as the Title I students, in the comparison 
group model it is from data of a comparable preselected group of 
students not receiving Title I services, and in the special regression 
model it is derived from data of students scoring above a 
predetermined Title I participation cutoff score and applying a 
statistical procedure to create a no treatment expectation for Title I 
participants. 

Each of the models allows for the reporting of Title I 
achievement gain through a common reporting metric known as the Normal 
Curve Equivalent (NCE), thus facilitating data aggregation and 
cross-site comparisons. Under proposed regulations current Federal 
plans are to require the districts to implement any of the three 
evaluation models for their Title I programs or an alternative model 
approved by USOE and the SEA, (Proposed regulations Part 116a ,31 of 
Title 45), Any alternative model proposed must develop a performance 
expectation for Title I students in the absence of Title I services 
and be able to report in the common reporting metric. 

Full utilization of the evaluation models in all school districts 
and the reporting of results in the common metric Is scheduled for FY 
1980 Title I programs, after the publication of final regulations in 
this area* However, states and districts have been gradually 
preparing themselves to implement model requl ements since 1976, with 
significant assistance from the USOE funded Technical Assistance 
Centers (TACs) . 

As Wi.th the creation of the mandated evaluation models the 
creation of the TACs grew out of the 1974 Education Amendments, 
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USOE was required by the law to provide technical assistance to 
implement the evaluation models (PL 93-380, . Section 151, Part E). 
Initially, however, no USOE monies were designated for such purposes. 
Under Congressional pressure USOE agreed in 1976 to cut 10 million 
dollars from a national independent evaluation of Title I impacts and 
to use those funds to create regional TACs which would assist states 
and LEAs in conducting cheir Title I evaluations. Contracts for 
operation of the TACs were awarded by USOE through competetive bid in 
the fall of 1976 and TACs continue to operate today. 

The main purpose of the TACs has been to assist states and LEAs 
in meeting the new Title I evaluation and reporting requirements. 
TACs function in response to specific grantee requests for assistance 
ana thus only operate to the extent they are perceived in the field as 
providing a useful advisory service. NTS as one of the regional TACs 
has developed a series of Title I evaluation workshops in response to 
grantee needs. The workshops address such issues as the steps to take 
in implementing each of the models, selecting the appropriate test 
instrument for measuring student achievement and the development of 
quality control procedures to help insure accurate data reporting. In 
addition, NTS staff respond to specific assistance requests from the 
field by telephone such as when a p.'.rfcular test can be administered 
without violating model assumptions or how to interpret a test 
publisher's norms tables. 

Where feasible and appropriate NTS staff have worked with 
grantees on the question of utilization of evaluative information from 
the models and other sources for state and local program decision 
making. Attempts to expand efforts in this area (such as through 
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holding a workshop on the topic of evaluation utilization) have thus 
far been stymied by USOE fears that such a TAG role exceeds USOE's 
mandate in this area i.e., that the authorized USOE tec'inical 
assistance role is largely restricted to "teaching the models." 

B. Implications For SRC Research Findings 

Two major conclusions seem to be emerging at this time concerning 
the impact of the models and TACs on the conditions observed in the 
SRC case studies. The first is that the most basic findings 
previously reported from these studies continue to hold true. Neither 
the models themselves nor the TACs have changed basic grantee 
attitudes toward evaluation utilization. An NTS coxieague who has 
conducted nearly 100 evaluation workshops as a TAG representative 
described the situation this way. 

"I do feel the LEA evaluations are better than they were. 
But, I don't feel that the LEAs are utilizing test scores 
any better than before. . .Nothing about the models or TIERs 
prevents them from making more frequent and better use of 
evaluative information. It is their lack of knowledge on 
what to do with data and the lack of help from the present 
TAG effort which perpetuates the LEAs "non-use" of 
evaluative data for local decision making and planning." 
Jane David's 1978 study of evaluation utilization at the LEA 
level reached similar conclusions. Dr. David found that local 
evaluations were not used either as a basis for judging the 
effectiveness of Title I programs or a guide for program 
decision-making (David p. 37). Concerning the question of the impact 
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of the evaluation models on local utilization, she states the 
following : 

"The models address only the "symptoms," that is, 
technical weaknesses of the outcome measures and 
procedures for data collection and analysis. I suggest 
that this approach. . .cannot by itself affect local use 
of evaluation." (David p, 42). 
To these observations this additional insight from the SRC case 
studies can be added. Since most states and local districts were 
observed to be structuring their administrative activities around 
Federal expectations, the recent Federal emphases on grantee 
evaluation technique and reporting requirements without similar 
attention to the utilization question should act to reinforce grantee 
feelings that their Title I evaluation responsibilities consist solely 
of providing data for other peoples use. The major change in 
Federal performance expectations emanating from TIERs is to expect 
grantees to pay considerably more attention to insuring that the data 
submitted to USOE is technically sound. This is not an insignificant 
change in Federal performance expectations but neither is it one which 
should by itself alter the conditions observed in the case studies 
concerning evaluation utilization. 

While there is no reason to believe that the existence of the 
evaluation models and TACs have thus fa. changed state and local 
attitudes toward data utilization, there is reason to hypothesize that 
this situation could change markedly in the coming few years. Both 
the creation of the models and the TACs have put forces in motion 
which may lead to increased utilization of evaluation in -innation for 
state and local decision-making. The most basic of these is the 
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heightened awareness of grantees to the issue of evaluation. The SRC 
case studies observed that States and LEAs generally devoted little 
administrative attention to evaluation beyond making sure that pretest 
and posttest scores were submitted for all Title I students. The 
imposition of the models, with their comparatively detailed aai 
rigorous technical requirements has drastically changed that 
situation. State and LEA administrators are being sensitized, both by 
the new requirements and the TACs , to the fact that evaluation is a 
serious and complex responsibility, consisting of considerably more 
than "entering some numbers on a report form." It is not insignificant 
that in all three of the districts observed as having the most 
promising Title I evaluation approaches. Title I officials were made 
cognizant by central LEA personnel of the importance and seriousness of 
the program evaluation task. This message has reached many more Title 
I administrators in recent months as a result of all the "noise" 
associated with the imposition of the new evaluation system. 

Moving from a grantee awareness that evaluation is a serious 
business to a realization that it can and should be used to help 
improve their own programs is admittedly a large step. The former, 
however, is probably a necessary condition for the latter to occur. 
What is then needed is an external agent to encourage and aid grantees 
in taking that step when district conditions are not by themselves 
fortuitous in that regard. There is evidence that TACs are willing 
and anxious to take on such a new responsibility and that recent 
changes in the Title I legislation may result in USOE allowing them to 
do so. 
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Federal interest in the question of evaluation utilization is 
reflected in the 1978 Education Amendments where for the first time the 
issue is specifically addressed in the legislation. Section 124 of 
Pulbic Uw 95-561 (Part G) states that, "...the results of the 
evaluations will be utilized in plannii.g for and improving projects and 
activities carried out under this title in subsequent years." Draft 
Federal regulations on the specific issue of evaluation utilization for 
program improvement are expected shortly. Recently, a TAG technical 
subconmittee on evaluation utilization (made up chiefly of TAG 
representatives from the various regions who make recommendations to 
USOE concerning TIERS), has been specifically encouraged to investigate 
the question of a TAG role in evaluation utilization that goes beyond 
the implementation of the three mandated evaluation models. While it 
is impossible to say at this time whether the TAG role will in fact be 
expanded to directly tackle the utilization question the recent 
activities of Congress and USOE have served to considerably strengthen 
such a possibility. 

Summarizing briefly the existence of the mandated evaluation 
models and the TAGs have not thus far affected the most basic 
conclusions from the SRG case studies regarding grantee approaches to 
Title I evaluation. Recent evidence indicates that evaluation is still 
viewed largely as data to be collected for someone &],ses use, and 
grantee attention to evaluation continues to focus upon issues of 
technique rather than usage. However, both the models and the TAGs 
have helped set forces in motion which may soon change that picture. 
The evaluation models and TAGs have heightened grantee awareness about 
the evaluation question, a necessary precursor for significant change. 
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And TAGS have themselves become a direct force for change by pushing 
for a larger role in helping grantees use evaluative data for their own 
purposes. When combined with a new Federal emphasis in this area as 
reflected in the 1978 Education Amendments it seems reasonable to 
conclude that circumstances now appear ripe for beginning to alter the 
persistent conditions observed in the SRC case studies concerning 
grantee utilization of Title I evaluation data. 
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